Linear mode connectivity describes a phenomenon observed for neural networks where every linear interpolation between two trained neural networks has roughly the same loss. In other words, there is a rather flat line in the loss surface connecting the two networks. We know that linear mode connectivity (LMC) doesn’t hold for two independently trained models. … Continue reading Layer-Wise Linear Mode Connectivity